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ABSTRACT 



A discussion of survey methodology for investigating second 
language programs and instruction examines two methods : oral interviews and 
written questionnaires. Each method is defined, and variations are explored. 
For interviews, this includes individual , group, and telephone interviews . 

For questionnaires , this includes self -administered and group-administered 
forms. Uses of information resulting from surveys are noted, including 
curriculum development, analysis of program needs, program evaluation, and 
other forms of language program research such as student characteristics and 
attitudes. Six steps in a survey research project are specified: planning the 
project; designing the survey instrument; gathering and compiling survey 
information; statistical analysis of results; logical analysis of results; 
and reporting results. Functions of the survey, which can stand alone or be 
combined in either questionnaire or interview, are discussed. They include 
collection of demographic information, investigation of opinion, 
self-ratings, gathering of information about judgments, rankings, and 
Q-sorts. Guidelines for writing good survey questions and effective 
questionnaires and for avoiding survey pitfalls are offered, with examples. 
Suggestions are also made for increasing mail survey return rate and 
tabulating and analyzing both open- and closed-ended responses. Contains 34 
references. (MSE) 
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Designing Surveys 
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James Dean Brown 

University of Hawaii at Manoa 




A language survey is any procedure which gathers information about partici- 
pants’ characteristics and views on die nature of language or language learning 
Uirough the use of oral interviews or written questionnaires. In this paper, I will 
begin by examining die different types of surveys, including both interviews 
and questionnaires. I will dien discuss how language surveys can be used for 
language curriculum development and research. I will continue by exploring 
die specific functions that surveys can perform. The next two secdons will 
provide a set of guidelines for wridng effecdve survey questions along with 
discussion of die issues involved in achieving a good return rate. I will then 
present a brief section on analyzing survey results and will end the paper with 
suggestions for further reading. 



Types of Surveys 

As shown in Table 1, surveys can take two basic forms: interviews (including 
individual, group, and telephone interviews) and quesdonnaires (including self- 
administered or group administered quesdonnaires). Each of these categories 
and subcategories iiieiits a bit more discussion. 
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Table 1 

Types of Survey Procedures 



Type of Survey 


Procedures 


Intei’views 


Individual 




Group 




Telephone 


Questionnaires 


Self- administered 




Group-administered 



Interviews 

Interviews are procedures for gathering information orally in specific, planned 
categories (if tlie interview is well structured in advance) as well as information 
that was not anticipated at die outset. Interviews can be conducted in three 
ways: individually, in groups, or by telephone. 

Individual interviews are interviews conducted widi participants on a one-to- 
one bases. Sucli interviews gadier information privately, so a certain level of con- 
fidentiality can be maintained. As a result, individual interviews have die advan- 
tage (over group intei’views) of collecting something close to die true views of die 
participants. However, such interviews also have the disadvantage of being rela- 
tively tinie-consuniing. As a result, individual interviews may be most effective for 
finding out wliich questions, views, or issues are worthy of later follow-up study 
using more staictured group interviews or questionnaires, both of which can be 
administered niucli more efficiendy than individual interviews. 

Group interviews, which are interviews conducted in a group or meeting for- 
mat, can be used to circumvent the time-consuming nature of individual inter- 
views. However, group interviews also have the disadvantage of lacking confi- 
dentiality. For personal, political, or emotional reasons, the opinions expressed 
when participants are in a group interview may be quite different from the views 
expressed when they are interviewed individually and confidentially. 

Telephone interviews sometimes provide a useful compromise in that they 
are confidential and somewhat less time-consuming (saving the time neces- 
sary to travel to the participants). However, there are also several disadvan- 
tages that must be considered in using this type of interview: (a) in a tele- 
phone interview, some information may be lost because the interviewer can- 
not see the facial expressions, gestures, and surroundings of the participants; 
(b) a telephone survey also precludes the possibility of showing participants 
anything during the interview; and (c) at least in the United States, many 
people have become wary of telephone sales promotions and surveys and 
may not cooperate. 




3 



Designing Surveys for Language Programs 



111 



Questionnaires 

The various types of intemews can clearly be used to explore the questions, 
topics, views, opinions, etc. of participants in a particular language teaching 
institution. However, if tliese points are worth pursuing on a larger scale, ques- 
tionnaires may prove much more efficient. A questionnaire is any written in- 
stalment that presents participants with a series of questions or statements to 
which they should react either by selecting from existing possibilities or writing 
out their answers. Questionnaires are particularly efficient for gathering infor- 
mation on a large scale. For example, an interview with a few of the participants 
in a language program might be helpful in designing a questionnaire, but the 
questionnaire itself would be much more practical to administer to ail of the 
participants. Most questionnaires are the self-administered type, A self-adminis- 
tered questionnaire is usually mailed out and filled out by the participants in 
tlieir own homes or offices (i.e., tliey are self-administered) and then returned 
by mail. Self-administered questionnaires have three potential problems: (a) 
they often have a veiy low return rate, (b) they must be completely self-ex- 
pianatoiy because further clarification is not possible, and (c) they are adminis- 
tered under conditions that are unknown to the survey designers, A group- 
administered questionnaire is any questionnaire which is administered to par- 
ticipants while they are in class or at a meeting. For example, if I wanted to 
suivey a particular group of students, I could go to their classes and ask their 
teachers to let me use die 15 to 20 minutes necessary for the students to fill out 
my questionnaire. By using such a strategy, I would solve all diree of the prob- 
lems listed in die previous paragraph for the self-administered quesdonnaires: 
(a) the students will be a capdve audience and will feel obliged to fill out the 
questionnaire (thus giving me a high return rate), (b) I can be present to explain 
any ambiguities as they arise, and (c) I will know exacdy what conditions ex- 
isted when die questionnaires were filled out. 



Uses for Language Surveys 

Sui-veys are most commonly used in language education for curriculum devel- 
opment and for research. Language programs provide a service to students who 
want to learn the language in question. If die program is interested in what 
diese clients tliink, then some form of curriculum analysis and research will 
probably be desirable because the best way to find out what people are think- 
ing about any aspect of a language program is to ask them. 

Surveys for Curriculum Development 

The two elements of curriculum development wherein survey research is most 
useful are needs analysis and program evaluation (for more details on what all 
of these elements are, see Brown, 1989, 1995a). While surveys can be used to 
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investigate many elements of a curriculum (including objectives, tests, materi- 
als, and teaching), it is needs analysis and program evaluation that most often 
require sui-veys. 

Needs analysis. Needs analysis in language programs is often tliought of 
astlie saidy of the language forms students will need to use in tlie target lan- 
guage when they actually try to communicate. The focus is usually on the 
learners, and tlieir needs are most often stated as linguistic needs. In fact, focus- 
ing on tlie learners in a needs analysis seems reasonable. After all, they are the 
clients; and tlie needs of the clients should be met. However, this view may be 
shortsighted because teachers, administrators, parents, employers, institutions, 
and even nations may also have needs that should not be overlooked in doing 
a needs analysis for a particular language program. Needs analysis should prob- 
ably focus on the language needs of students, but must recognize that students 
also have other needs, called in Brown (1995a) “situational needs.” Situational 
needs include all of die learners needs as human beings. This means that the 
definition of needs analysis should be fairly broad. For instance, needs analysis 
might be defined as the systematic collection and analysis of all relevant infor- 
mation necessaiy to meet the language learning needs of die students within 
the context of the particular institutions involved in the learning and teacliing 
situation. Since needs analysis relies on input from a variety of different groups 
about the saidents’ linguistic and situational needs, survey projects may prove 
particularly useful. In fact, in gatiiering information from the various groups of 
people involved, survey interviews and questionnaires may become inevitable. 
After all, one of the most positive features of survey research is that it can be 
used to efficiently gather a wide variety of different types of information from a 
wide assortment of sources. 

Program evaluation. Program evaluation might be defined as die systematic 
collection and analysis of all relevant information necessary to promote the 
improvement of the curriculum and assess its effectiveness within the context 
of the particular institutions involved. Since this definition is very similar to the 
one given above for needs analysis, the reader might correctly conclude that I 
view the program evaluation process as a kind of ongoing needs assessment. 
That is true. However, program evaluation is typically based on considerably 
more and clearer information tlian needs assessment. While needs analysis is 
usually done in the early stages of curriculum development and must depend 
on intei-view procedures, questionnaires, linguistic analyses, conjecture, and a 
good deal of professional judgment, program evaluation can take advantage of 
all of the above information and tools to assess the effectiveness of a program 
but can also utilize any other information gathered while developing program 
objectives, developing and analyzing tests, creating materials, and/or teaching 
the curriculum. Program evaluation^ then, will be defined here as the ongoing 
process of information gathering, analysis, and synthesis, the entire purpose of 
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which is to constantly improve each element of a curriculum on the basis of 
what is known about all of the other elements, separately as well as collec- 
tively. Thus, survey research is a caicial part of program evaluation just as it is 
for needs analysis and for precisely the same reasons. While survey projects 
used in curriculum development generally remain in-house, some find their 
way into print (for examples, see Kimzin & Proctor, 1986; or Richterich & 
Chancerel, 1978). 

Suweys for Language Research 

Surveys are also useful for other types of research not directly related to curricu- 
lum development. For example, surveys have been used to investigate the char- 
acteristics of graduate level teacher-training programs (Ochsnef, 1980), or the 
opinions and attitudes of members of a national professional organization (Brown, 
Knowles, Murray, Neu, & Violand-Sanchez, 1992). Of all the available research 
methods, survey research may be tire most practical and easy to do because it 
relies heavily on common sense and less on complex statistics. In fact, survey 
results are most often reported as simple percentages and averages. Since lan- 
guage survey research can be used to describe, explore, or explain physical 
characteristics, phenomena, behaviors, attitudes, etc., tliere is no shortage of 
useful questions that can be answered by doing survey research. Given that 
people’s ideas, attitudes, opinions, etc. change over time, the possibilities for 
doing language survey research seem inexliaustible because any research done 
today might usefully be repeated in five or ten years to see if tlie participants’ 
ideas, attitudes, opinions, etc. have changed over time. 



Steps in Survey Projects 

The main problem in setting up a survey project is to make sure that it is 
planned carefully in a step-by-step manner so the project can be conducted 
systematically and effectively. Stacey and Moyer (1982) suggest the following 
ten steps for die consti\iction of surveys: 

1. Specifying survey objectives and researcli questions. 

2. Reviewing the literature. 

3. Defining abstract concepts. 

4. Selecting question formats. 

5. Selecting die statistical analysis. 

6. Writing the sui*vey questions. 

7. Ordering the questions in die survey. 

8. Adjusting die physical appearance of the survey. 

9. Preparing the cover letter and instructions. 

10. Validating the sui*vey. 
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These ten steps are fine if the only concern is with creating a survey instrument. 
However, I feel that a language survey instalment should be viewed as part of 
a larger survey project, one which involves the following steps: 

A. Planning the survey project. 

B. Designing the survey instrument. 

C. Gathering and compiling the survey information, 

D. Analyzing tlie survey statistically, 

E. Analyzing tlie survey logically, 

F. Reporting the results of a survey. 

If each of these six steps is carefully carried out (along with the ten steps for 
constaicting a sui-vey proposed by Stacey and Moyer (1982) under point B of 
my list), any sui*vey project will stand a much better chance of success. 



Functions of Surveys 

Anyone designing a survey instrument must realize tliat surveys can function in 
different ways. For instance, surveys can function as: biodata surveys, opinion 
surveys, self-ratings, judgmental ratings, rankings, or Q-sorts. Obviously, some 
or aU of these functions can be combined in one interview or questionnaire. 

Biodata surveys are used to elicit biographical background information about 
tlie participants, including things like the participant’s age, place of birth, sex, 
marital status, occupation, number of years of language study, years of educa- 
tion, major field of study, and salary. 

Opinion surveys are designed to uncover the opinions and attitudes of the 
participants about specific issues. For instance, questions might be developed 
to find out what a group of students think (i.e,, tlieir opinions) about tlie goals 
and objectives of a particular language program as well as about its tests, mate- 
rials, teacliing, administration, etc. 

Self-ratings require the participants to rate their own abilities, interest levels, 
motivations, etc. Such ratings reveal information about the self-image of the 
individuals involved. 

Judgments can be used to obtain the views of participants about various 
aspects of language learning. Judgments can be obtained either by asking par- 
ticipants to briefly write their views down or by using a more structured type of 
question. For example, saidents might be asked to judge the effectiveness of a 
language program’s objectives and materials in terms of how useful they are, 
how necessary tliey are to future language use, or how easy they are to learn. 
The students would simply be providing their judgments. 

Rankings require the participants to order some set of words, or phrases, 
from most important to least important, or most useful to least useful, etc. For 
instance, srtidents might be asked to rank a list of the 15 types of classroom 
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activities used in a particular course from most enjoyable to least enjoyable by 
numbering them 1 to 15. 

Q-sorts combine attributes of one of tlie earlier survey types with the rank- 
ing procedures discussed immediately above. Tliis combination is accomplished 
by: (a) having the participants give their views, attitudes, opinions, self-ratings, 
and/or judgments; (b) then asking the participants to rank tlie issues in terms 
of how important they think each is. While survey instruments can be designed 
to perform any one of the six functions discussed above, they are seldom so 
unidimensional; surveys are most often developed to accomplish a combina- 
tion of functions. 



Guidelines for Writing Good Survey Questions 

Once decisions have been made about which functions to use in a survey (or 
which combinations of functions), the actual questions must be written. Writ- 
ing good questions can be surprisingly difficult. Guidelines for writing good 
survey questions are given in Table 2. Overall, the goal in writing survey ques- 
tions should be to make tliem so clear and precise that the participants will 
know exactly what is being asked of them. 



Table 2 

Things to avoid in writing good survey questions 



Avoid: 

1. Overly long questions 

2. Unclear or ambiguous questions 

3. Negative questions 

4. Incomplete questions 

5. Overlapping choices in questions 

6. Questions across two pages 

7. Double barreled questions 

8. Loaded word questions 

9. Prestige questions 

10. Embarrassing questions 

12. Biased questions 

13. Questions at tlie wrong level of language 

14. Questions that respondents are incompetent to answer 

15. Assuming that everyone has an answer to all questions 

16. Making respondents answer questions that don’t apply. 

17. Using irrelevant questions 

18. Writing superfluous information into questions. 
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Notice tliat Table 2 is cast in negative terms because it is a list of tilings tliat 
should be avoided in writing survey good survey questions. To begin with; (1) 
overly long questions, (2) unclear or ambiguous questions, (3) negative ques- 
tions, and (4) incomplete questions should all be avoided. Overlapping choices 
(5) should also be avoided. The issue of overlapping choices arises when op- 
tions are not mutually exclusive. In the following example, die options are 
overlapping in die sense that Santa Ana High School is located in die city of 
Santa Ana and in Orange county so the opdons are not mutually exclusive: 

The best location for the new foreign languages building would be: 

^In Orange County 

^In the city of Santa Ana 

In Santa Ana High School 

Other: 

Quesdons should also be typed so they are located in one compact place on 
a single page (6) so diat participants will not miss part of die quesdon. Double- 
barreled quesdons (7) arise when a quesdon is really asking more than one 
question. For instance, the following example contains three quesdons that 
could be asked separately (one about educadon spending, one about military 
spending, and one about language educadon spending): 

Tlie United States should spend more on educadon (especially 
language education) and less on die military. 

Agree Disagree Don’t kno w 

Loaded words (8) are words that suggest an automatic posidve or negative 
response because diey are emodonally charged. For instance, in the following 
example, the word obviously would be a loaded word: 

Spanish is obviously die language to study if you live in the 
Soudiwestern United States. 

Agree Disagree Don’t kno w 

Leading quesdons (9), namely, quesdons that overdy encourage participants to 
answer in a certain way, should also be avoided. It is also important to avoid 
quesdons diat may contain issues of prestige (10). For example, when asked, some 
people will claim that diey read more than they do or that they speak a language 
better dian diey do because diat is the prestigious way to answer. Embarrassing 
quesdons (11), or quesdons diat a particular group of participants miglit find em- 
barrassing (e.g., quesdons containing swear words) should be avoided, as should 
quesdons that show prejudice or bias (12) against a particular group of people. 

In addition, quesdons at the wrong level of language (13) and questions diat 
participants are incompetent to answer (14) should be avoided. And, as matters 
of sound sui-vey development policy, it is probably best not to (15) assume that 




9 



Designing Surveys for Language Programs 



117 



eveiyone has an answer to all questions, (l6) make participants answer ques- 
tions that don’t apply to them, (17) use irrelevant questions, or (18) write super- 
fluous information in questions. 



Guidelines for Producing Effective Questionnaires 

Writing good questions, as covered in the previous section, is just one of die 
issues involved in producing sound surveys. Many others are covered in Brown 
(unpublished ms.) 



Table 3 

Guidelines for Producing Effective Questionnaires 



1. Write good questions (see Table 2) 

2. Order the questions rationally 

3. Format the questionnaire for clarity 

4. Write clear directions 

5. Edit carefully 



For questionnaire surveys. Table 3 suggests diat die ordering of questions is also 
impoitant. In order to make a questionnaire clear to the participants, it may help to 
mtionally gioup the questions by dieir type, function, format, topic, or some com- 
bination of diese factors. (For more ideas on stmcturing surveys, see Gallup, 1947), 
Formatting the questionnaire for clarity is anodier important issue, which 
may involve effective use of spacing, type faces, graphics, highlighting, etc. 
Writing clear directions is another CRicial issue to diink about because unclear 
directions can confuse and discourage participants. Last but not least, it is nec- 
essaiy to use careful editing, perhaps including getting others to edit the ques- 
tionnaire, piloting it with a few saidents, and doing very careful final editing. 



Guidelines for Increasing Mail Survey Return Rates 

There are also a number of ideas that might help in getting a relatively high 
return rate, especially for those who are using a mail survey, where a question- 
naire is mailed out and paiticipants are asked to send it back after filling it out. 
First, a cover letter explaining the background and importance of die question- 
naire may help. Participants are also more likely to return a short questionnaire 
that is easy to fill out than to return a long and difficult one. If possible, some 
sort of incentive should be offered to participants for answering the question- 
naire (e g., a monetaiy reward, or a copy of die study that will result from the 
questionnaire). Providing return postage and a self-addressed envelope will 
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also encourage participants to return die questionnaire. It is also a good idea to 
put the return address somewhere else on die questionnaire so it can be sent 
back even if the participant loses the envelope. 

For participants who do not return die questionnaire, a follow-up letter or 
phone reminder may encourage them to complete die task. The timing of the 
mailing may also be significant. For instance, if a questionnaire mailing goes out 
late in die school year or arrives during the summer break, it is unreasonable to 
expect academics to respond quickly in great numbers. 



Tabulating and Analyzing Survey Results 

Once the suivey results have been collected, some system must be worked out 
for tabulating and analyzing all of the data. The system will differ for closed- 
response questions and open-response questions (bodi of wliich may be found 
on die same questionnaire). 

Closed Responses 

Closed-response questions are diose which offer die participants options from 
wliich to select dieir answers. This category includes Likert scales (the 1 to 5 type), 
yes/no questions, or any odier select-a-response type question. Such questions 
can be analyzed widi stoightfoiward averages or percentages of people selecting 
each option, or bodi. For die statistically initiated, standard deviations and chi- 
squared statistics may prove useful, but the averages and percentages are usually 
sufficient for finding interesting patterns in closed-response survey results. 

In addition, it is often a good idea to check the reliability of a survey instru- 
ment — especially if it is being used to make important decisions or to do pub- 
lishable research. Given diat there are often weighted scales on questionnaires, 
die Cronbach alpha reliability estimate is probably the best to use. Cronbach 
alpha is available for use on MS-DOS system computers in the SPSS/PC7+'^^ 
(1990) or^KS’TMF^ (Wilkinson, 1988) software, or it can be calculated by hand 
using the formula found in Brown (1995b). 

If there are subscales involved in the survey, it may be a good idea to exam- 
ine die reliability of each subscale as well as the combination of all closed- 
response subscales taken together. 

Open Resp07ises 

Open-response questions are questions that require the participant to produce 
an answer, in either an oral or written form. If die responses are oral, they 
should be taped in the process of gathering the responses. It is usually neces- 
sary or at least advisable to transcribe the tapes to a written form. 

Even if die open responses are written by die participants, I have usually found 
it necessaiy to enter diem into computer files for easier analysis. I then do com- 
puter analysis of these files using a software package like Bi^htWritef^ (Que 
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Software, 1990) to get frequency counts of the words in the files. Sometimes, as 
part of tlie analysis, I use die search function in my word processor to searcli for 
die most commonly used words and read how the respondents used diem. 

Widi or widiout the technological boost diat computers provide, die open 
responses should all be read, analyzed, sorted, and summarized in some way. 
This process may involve simply putting diem on cards and getting down on 
the floor to sort them out, or may involve higli teclinology. Eidier way, the 
human bi*ain ultimately ends up doing the analysis and interpreting the results. 
For examples of interpretations of such open-response questions, see Brown 
(1992a, 1992c, 1992d, & 1992e). 

Suggestions for Further Reading 

A number of different topics liave been covered in this chapter. Naturally, each 
of diese topics is covered in more detail elsewhere. For instance, more informa- 
tion on the different types, uses, and functions of surveys, as well as ideas for 
writing good suivey questions and getting good return rates can be found in 
Babbie (1973); Bailey (1982); Brown (unpublished); Oppenheim (1966); or Stacey 
and Moyer (1982). Examples of how surveys have been used for language 
curriculum development (especially for needs analysis and program evaluation) 
are available in Brown (1995a); Kimzin and Proctor (1986); Loschky, Stanley, 
Cunha, and Singh (1987); Richterich and Chancerel (1978); and Weaver, Pickett, 
Kill, and Cook, J. (1987). 

Examples of how suiveys have been used in language research can be found 
in Bailey and Brown (1995); Brown (1992a, 1992c, 1992d, 1992e, and 1995a); 
Brown, Knowles, Murray, Neu, and Violand-Hainer (1991, 1992); Johns (1981); 
Miller (1995); and Ochsner (1980). 

More details on using computers for program level decision making and 
research can be found in Brown (1992b). 

Additional information on the statistics necessary to analyze closed-response 
survey results, see Brown (1988); Butler (1985); Hatch and Lazaraton (199D; 
Seliger and Shohamy (1989); and Woods, Fletcher, and Hughes (1986). 

More details on increasing the reliability and validity of the open-response 
analytic processes are available from Agar (1986); Davis (1992); or Kirk and 
Miller (1986). 

Survey interviews and questionnaires provide seemingly simple and straight- 
foiward tools tliat can be used effectively for curriculum development or other 
types of research. However, language professionals who actiially use these tools 
often find tliat they are not as simple as they seem. Hopefully, this paper has 
provided much of the information necessary to effectively plan and conduct 
surveys, or at least pointed the reader to sources that can help them use surveys 
responsibly and professionally. 
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